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ABSTRACT 


In 2009, the Naval Oceanographic Office was tasked with developing a product that uses 
forecasted meteorological conditions and historical pirate incidents to predict locations 
conducive to pirate activity in the Somali Basin Region and the Gulf of Aden. This 
resulted in the development of the Piracy Performance Surface (PPS) model, whose 
outputs are briefed daily to the Commander of the United States Naval Forces Central 
Command and Combined Maritime Forces in Bahrain. The Next-generation PPS (PPSN) 
model uses simulation to provide as output, a forecast of relative pirate presence 
probability over time. Effective March 1, 2011, the name of PPSN has been changed to 
the Pirate Attack Risk Surface (PARS) model. 


This research includes interviews with counter-piracy forces that led to 
recommended changes in the PPSN model. In addition, using robust and realistic 
experimental designs, this research identifies the significant intelligence factors of the 
PPSN model. This gathered information is being used to refine these input variables to 
achieve maximum performance of the PPSN model. This research also unveiled input 
variables that are influential in the computing memory requirements and program 
runtime. This information is being used to focus efforts on setting these variables to 
realistic levels without sacrificing the model's efficiency and effectiveness. Finally, the 
results of this thesis allow for quick turnaround of updates to the PPSN model in response 


to gathered intelligence. 
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EXECUTIVE SUMMARY 


The instability and lack of government in Somalia have caused a rapid increase in pirate 
activity within the Somali Basin Region (SBR) and the Gulf of Aden (GOA). Due to this 
increase, maritime security and stability in these regions are severely threatened. Somali 
pirate attacks have taken the lives of innocent people, held sea-going persons hostage, 
and cost the world's economy billions of dollars annually (Bowden, Hurlburt, Aloyo, 
Marts, & Lee, 2010, pp. 12). For this reason, President Barack Obama initiated an 
executive order on April 12, 2010 to deal with the threat of piracy (Executive Order No. 
13,536, 2010). President Obama stated that the collapse of security and the continuation 
of violence in the Somali region in the form of piracy and armed robbery on the high seas 
are a threat to the United States’ (U.S.) national security and foreign policy of the United 


States and must be dealt with accordingly. 


U.S. Naval Forces Central Command (NAVCENT) Commander Vice Admiral 
Mark Fox and U.S. Sixth Fleet Commander Vice Admiral Harry Harris, Jr. have been 
tasked with finding a solution to deter, control, and ultimately eliminate pirating within 
the GOA and SBR within the U.S. Fifth and Sixth Fleets’ operating waters by returning 
safety, stability, and peace to these regions. Bahrain, being strategically located within 
the Middle East, has been established as the U.S. Fifth Fleet and Combined Maritime 
Forces (CMF) Headquarters for the Combined Maritime Forces, the North Atlantic 
Treaty Organization (NATO), and the European Union (EU); meanwhile, the U.S. Sixth 
Fleet headquarters resides in Naples, Italy. 


To combat the on-going concerns with piracy, Naval Oceanographic Office 
(NAVO) researchers were tasked by Rear Admiral Titley, Oceanographer and Navigator 
of the Navy, to assist in the fight against piracy. This tasking occurred just days after the 
MV Maersk Alabama, a U. S.-flagged ship, was hijacked by Somali pirates in the Indian 
Ocean on April 18, 2009. The Piracy Performance Surface (PPS) model was completed 
approximately two weeks after this incident, and has undergone a few revisions since its 
development. The success of the PPS model led to the Naval Meteorology and 


Oceanography Command leadership to devote efforts to pursue a more advanced anti- 


XVI 


piracy model, the Next-generation PPS (PPSN). On March 1, 2011, the PPSN model's 
name changed to the Pirate Attack Risk Surface (PARS). Dr. James Hansen of the Naval 
Research Laboratory (NRL) in Monterey, California, was tasked with developing this 


next generation probabilistic forecasting tool. 


The PPSN model is a Monte Carlo probabilistic forecasting tool that takes as 
input environmental conditions, pirate concept of operations, and gathered intelligence to 
produce as output the forecast of pirate presence probability as a function of latitude, 
longitude, and time. The model simulates pirate action groups (PAGs), where a PAG is a 
group of pirate dhows that are attached to a mother dhow (a larger vessel) that is typically 
used as a logistical hub for extending distances from land and mission duration. The 
PPSN outputs a forecast of relative pirate presence probability over a 72-hour forecast 
time period in 0.2-hour time steps with a probability surface generated at 12-hour time 
steps (See Figure 1). As of the date of this thesis, the PPSN model is the only known 
Navy product that couples METOC and INTEL. 


25] 
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4 50 5 sa 6 35 40 45 50 55 60 65 
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Figure 1. The PPSN model output displaying the forecast of relative pirate presence 
probability with the 24-hour forecast (left) and the 72-hour forecast (right). Red 
corresponds to the highest probability forecast of pirate location. 


This thesis research includes visits and interviews with users of PPS and other 
counter-piracy forces to understand the operators' needs and to gain intelligence on the 
most current pirate CONOPS within the SBR and GOA. This gathered intelligence led to 


recommended changes in the PPSN code to include the addition of PAG search patterns, 


XVill 


the implementation of PAG operating boxes and their mapping, and the understanding of 


pirate terminology used by military forces combating piracy. 


Key findings this research addresses are: (1) which input parameters can be 
adjusted to decrease run time and computing memory requirements, (2) which input 
parameters interact most strongly with METOC, (3) which input parameters are the most 
influential to the PPSN output, and (4) what improvements can be made to enhance the 


PPSN model. 


Three key input parameters that impact the computer memory requirement, which 
are also highly correlated with runtime, are the PAG's mission length, the number of 
PAG replications in the simulation, and the number of PAGs. The analysis by the author 
concludes that the values inputted for each of these variables, in particular, the PAG's 
mission length, should be considered with the utmost care as their associated values 


greatly impact the forecast of relative pirate presence probability. 


The search pattern the PAGs use when attempting to locate a target vessel of 
opportunity is the most significant factor in the PPSN model. Operators and military 
members combating piracy should attempt to acquire intelligence on the specific search 


pattern that PAGs use in the SBR and GOA. 


The probability that PAGs have meteorology and oceanography information prior 
to leaving their base location is highly influential and interacts with the most important 


variable, the PAG's search pattern. 


In addition, using systematic and robust experimental designs, this research made 
PPSN more robust and realistic by breaking the code in different ways, resulting in 


modifications by the developer to make PPSN better and more robust. 
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I. INTRODUCTION 


A. BACKGROUND AND MOTIVATION 


The instability and lack of government in Somalia have caused a rapid increase in 
pirate activity within the Somali Basin Region (SBR) and the Gulf of Aden (GOA). The 
SBR is the region that lies directly between Somalia and India in the Indian Ocean. The 
GOA 1s the body of water located between the Horn of Africa (HOA) and Yemen. Piracy 
in these regions has not only threatened the stability and safety of ocean-going vessels 
and their crew members, but has also cost the world’s economy over billions of dollars 
(Bowden, Hurlburt, Aloyo, Marts, & Lee, 2010, pp. 12).! For this reason, President 
Barack Obama initiated Executive Order 13536 on April 12, 2010 to deal with the threat 
of piracy. President Obama stated that the collapse of security and the continuation of 
violence in the Somali region in the form of piracy and armed robbery on the high seas is 
a threat to the United States’ national security and foreign policy of the United States and 


must be dealt with accordingly. 


Two U.S. Combatant Commands (COCOMSs) are tasked with solving the Somalia 
piracy problem, U.S. Africa Command (AFRICOM) under General Carter Ham (From 
http://www.africom.mil), U.S. Army, and Central Command under General James Mattis 
(From http://www.centcom.mil), U.S. Marine Corps (USMC). Both face an extremely 
complex situation with combating piracy that originates in the four distinct and 
independent political regions that make up Somalia (Figure 2), and finishes with 
hijacking vessels at sea. AFRICOM is currently conducting the African Partnership 
Station (APS) 2011 with 22 Affrican-partnered nations (LT S. Hall, personal 
communication, January 31, 2011).2..CENTCOM is responsible for the waters 
surrounding the SBR and GOA, and is conducting Maritime Security Operations (MSO) 


! The costs associated with piracy include ransom costs, insurance increases, re-directing commercial 
ships, additional security, the use of military vessels, legal jurisdiction for captured pirates, and 
organizations that help combat piracy. 


2 The author was forwarded this information through LT Stacey Hall. AFRICOM sent LT B. Le to 
present the African Partnership Station (APS) 2011 on behalf of AFRICOM (Africa Combatant Command) 
on February 2, 2011. 
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with Combined Task Force (CTF) 150 and CTF 151. As the reader can see, complexity 
exists in the Somali region and special attention must be exhibited by the Commanders of 
the two COCOMSs with a partnership for countering piracy operations on land and at sea. 
An additional layer of complexity exists with a separate U.S. fleet reporting to each 
COCOM and both fleets sharing the responsibility for the safety and stability of the 


waters surrounding Africa. 
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Figure 2. Map of the four political regions of Somalia used by the Combined Maritime 
Forces assessment team. Region one is defined as Somaliland; region two is 
defined as Puntland; region three is defined as Galmudug; and region four is 
Harakat al-Shabaab Mujahideen (From Wikipedia, 2011). 


U.S. Naval Forces Central Command (NAVCENT); Commander, U.S. Fifth Fleet 
and Commander, Combined Maritime Forces (CMF), Vice Admiral Mark Fox;? and 


3 For detailed information on Admiral Fox and the U.S. Fifth Fleet's mission statement and recent 
operations visit http://www.cusnc.navy.mil. 


Commander, U.S. Sixth Fleet, Vice Admiral Harry Harris, Jr.,4. have been tasked with 
finding a solution to deter, control, and ultimately eliminate pirating within the GOA and 
SBR within the U.S. Fifth and Sixth Fleets’ operating waters by returning safety, 
stability, and peace to approximately two million square miles of ocean water 
surrounding Somalia.” Bahrain, being strategically located within the Middle East, has 
been established as the U.S. Fifth Fleet and Combined Maritime Forces (CMF) 
headquarters for the Big Three: Combined Maritime Forces (comprised of 25 nations), 
the North Atlantic Treaty Organization (NATO), and the European Union (EU); 
meanwhile, the U.S. Sixth Fleet headquarters resides in Naples, Italy. LCDR K. Lutz® 
(personal communication, June 8, 2010) stated that other nations, such as China and 
India, are also working to defeat these opportunistic ocean-driven thieves, but these two 


nations conduct missions not in affiliation with the U.S., NATO, or the EU. 


In the 2010 National Security Strategy (NSS), President Obama stated that with 
the continued threat of unstable nations the U.S. military will constantly be called to 
provide military security and stability as part of globalization (President of the United 
States, 2010, pp. 12). Follow-on to the NSS is the 2011 National Military Strategy 
(NMS) guidance. In this paper, Admiral Mike Mullen, Chairman to the Joint Chiefs of 
Staff, stated that the Joint Forces will continue to address and combat the violent 
extremists that have sought a safe haven in the SBR and GOA regions as these extremists 
(pirates) remain a transnational security threat (CJCS, 2011). Senior military leaders are 
in a difficult position and General Mattis is a prime example. As CENTCOM 
Commander, General Mattis must divide his military assets (air, land, and sea) between 
the instability in the Middle East and the “rash of piracy” in the GOA and SBR (LCDR 
K. Lutz, personal communication, June 7, 2010). How to properly allocate military 


forces in order to sustain and ultimately eliminate piracy is a question that leading U.S. 


4 For detailed information on Admiral Harris and U.S. Sixth Fleet's mission statement and recent 
operations visit http://(www.naveur-navaf.navy.mil. 


> For detailed information on Central Command (CENTCOM) visit http://www.centcom.mil. 


6LCDR K. Lutz, UK, held the intelligence officer INTEL officer) position for the CMF assessment 
team in Bahrain during the author’s operations research experience tour. 
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military and civilian officials are looking to answer. This thesis seeks to support the 
mission by improving the tools available to decision makers to allocate forces, 


specifically forces combating piracy. 


The first operational piracy prediction model, Piracy Performance Surface (PPS), 
was developed in 2009 by the Naval Oceanographic Office (NAVO) at Stennis Space 
Center. The successes of PPS led to follow-on research for the development of a pirate 
probabilistic tool, the Next-generation PPS model (PPSN).’ This tool produces as output 
a forecast of relative pirate presence probability as a function of latitude and longitude 
over time. This thesis was conducted parallel to the development of the PPSN by the 
Naval Research Laboratory (NRL) Monterey and NAVO and explores the operational 
usefulness, product stability, and output consistency of PPSN. In addition, this research 
includes visits and interviews with users of the PPS model and other counter-piracy 
forces to understand their needs and pirate tactics, which led to recommended changes in 


PPSN. 


B. RESEARCH QUESTIONS 


This thesis addresses the following questions: 


1. What improvements can be made to the operational version of the PPSN 
model that will enhance the product’s output? 


2. Which PPSN input parameters can be adjusted to decrease computing 
memory requirements and run time? 


3. | Which PPSN input parameters interact most strongly with Meteorology and 
Oceanography (METOC) conditions? 


4. Which PPSN input parameters are the most influential in determining the 
distribution of relative risk of pirate presence in the PPSN” 


7 For continuity in this thesis, the name of the next-generation Piracy Performance Surface model 
(PPSN) will be used. Effective March 1, 2011 the name of the PPSN model was changed to the Pirate 
Attack Risk Surface model, PARS. 
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C, BENEFITS OF THE STUDY 


Using carefully designed simulation experiments, this research identifies the most 
important factors driving the PPSN output. This information can be used for refining 
intelligence inputs and the PPSN model development efforts to implement the most 
influential functionality. These results can also be used to optimize parameter selection to 
allow quick turnaround of updates to the product in response to INTEL, within 
computational constraints, and without sacrificing product quality. In addition, the 
information gathered in this thesis will be used to assist the military INTEL community 


(primarily in affiliation with the Big Three in Bahrain) with PPSN. 


D. THESIS ORGANIZATION 


Chapter II provides the reader with a literature brief that describes the basis for 
this thesis research. Chapter III describes the evolution of the efforts of the U.S. in the 
development of piracy models, and in particular the development of PPSN. Chapter IV 
defines the choice of independent variables for simulation modeling and the output 
metrics used to evaluate the forecast of relative pirate presence probability. Chapter V 
discusses each simulation experimental design. Chapter VI examines the results of the 
experiments, including multivariate correlation, stepwise regression, sensitivity analysis, 
and classification and regression trees. Chapter VII draws conclusions and makes follow- 


on recommendations. 
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I. LITERATURE REVIEW 


In this chapter, piracy will be defined and the following topics covered: costs and 
challenges created by piracy, current piracy concept of operations, other nations’ potential 


piracy tools, and the development of experimental designs. 


A. DEFINITIONS OF PIRACY 


Piracy has plagued the world for thousands of years. However, the U.S. 
government did not clearly define piracy for many years. In fact, the U.S. Congress did 
not take an aggressive role in defining piracy for the U.S. judicial system until December 
2010 (Mason, 2010). When it finally did, the definition took its origins from the 1982 
United Nations Convention on the Law of the Sea (UNCLOS) Article 101 that defines 
piracy as: 

(a) Any illegal acts of violence or detention, or any act of depredation, 


committed for private ends by the crew or passengers of a private ship or a 
private aircraft and directed: 


(1) On the high seas, against another ship or aircraft, or against 
persons or property on board such ship or aircraft; 


(11) Against a ship, aircraft, persons or property in a place outside 
the jurisdiction of any State; 


(b) Any act of voluntary participation in the operation of a ship or of an 
aircraft with knowledge of facts making it a pirate ship or aircraft; 


(c) Any act of inciting or of intentionally facilitating an act described in 

subparagraph (a) or (b). 

Other variations of the definition of piracy include the following: “an act of 
violence, or the threat of violence, against a ship in international waters (outside a state’s 
geographic sovereignty) with the intent to commit crimes such as theft, hijacking, 
murder, and kidnapping” (Kline, 2010, pp. 69); and “an act of boarding or attempting to 
board any ship with the apparent intent to commit theft or any other crime and with the 


apparent intent or capability to use force in furtherance of that act” (Chalk, 2008, pp. 3). 


Many variations of the definition of piracy exist, but the key to combating piracy 
begins by identifying the root cause of this multi-national threat (Mason, 2010). Two 


questions have come into the political arena (Goodman, 2009): 


(a) Did piracy develop because of illegal fishing in the economic 
exclusion zone (EEZ) of the SBR? 


Or 
(b) Did piracy develop from an unstable Somali government? 


The former CMF legal officer, CDR David Teasdale, United Kingdom (UK), 
stated that the UNCLOS has clearly defined piracy, but the real debate surrounding 
piracy is a “policy debate rather than a legal one" (personal communication, July 10, 
2010). The policy debate exists between the U.S., NATO, and the European Union. The 
UK Ambassador at the Security Council Debate on Somalia Piracy stated, "The United 
Kingdom believes that the best prospect of sustainable results will come from further 
enhancing the international community’s efforts to build the capacity of regional states to 
prosecute and imprison those responsible for acts of piracy" (Grant, 2010, para. 7). Grant 
(2010) later stated that he believes the root cause of piracy is a direct result of the 


instability of the Somali government. 


B. COSTS AND CHALLENGES CREATED BY PIRACY 


Chalk (2008) stated that the International Maritime Bureau (IMB) "estimates that 
piracy costs the shipping industry anywhere from $1 billion to $16 billion a year’ (pp. 
16). Paying the ransom money is more cost-effective than implementing protective 
measurers onboard, but this method is a quick fix to the ultimate problem. A Cooperative 
Strategy for the 21° Century Seapower recognizes the need for international partnership 
(Roughead, Conway, & Allen, 2007). “This strategy stresses an approach that integrates 
seapower with other elements of national power, as well as those of our friends and 
allies” (Roughead et al., 2007, para. 1). The commercial shipping community as well 
must continue to practice maritime awareness as modern day pirates are better trained 
and equipped with semi-automatic machine guns, varieties of rocket propelled grenades, 


and small arms (Best Management Practices [BMP3], 2010). 
3) 


C. CURRENT PIRACY CONCEPT OF OPERATIONS 


Pirate concept of operations (CONOPS), also known as pirate tactics, has been 
changing and evolving as the number of successful paid ransoms has increased. In April 
2010, Rear Admiral Peter Hudson, British EU Admiral, addressed a new pirate CONOPS 
swarming tactic that allowed pirate action groups to supersaturate the area once a target 
of opportunity had been identified (Childs, 2010). Pirates have also extended their 
operating area to 1,000 nautical miles from the shoreline because commercial vessels are 
no longer hugging the Kenya and Somali coastline (Figure 3). Admiral Hudson was also 
quoted in saying (Childs, 2010): 

Five years ago, the maximum range of attacks was 287 km (165 nautical 

miles). Recently a ship was hijacked 2,037 km (1,100 nautical miles) 


from the Somali coast - and only 926 km (500 miles) from the coast of 
India. 


Weather conditions also play a significant role in impacting pirate CONOPS. 
Historical METOC and INTEL reports have shown that pirate activity decreases in the 
SBR during the biannual monsoon season.® This decrease is due to the heightened wind 
and wave conditions that make the sea states unfavorable for pirating in the SBR. In 
2011, though, new pirate CONOPS have enabled the pirate action groups (PAGs) to 
operate in these adverse weather conditions. Admiral Fox stated (Fellman, 2011): 

The monsoon season once kept pirate skiffs ashore, but mother ships allow 


for year-round piracy. About eight of these “pirate action groups” are 
spread throughout the region, he said. 


8 See the Piracy Analysis and Warning Weekly (PAWW) Report (2010) for an example of this report. 
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Figure 3. This picture shows the extension of piracy missions from 2005 through 2011. 
In 2005, pirates were seen operating 165 nautical miles in the SBR and GOA, but 
in 2011, pirates are operating 1,300 nautical miles from the coastline in these 
areas (Somalia's states, regions, and districts) (From Childs, 2010). 





PAG 1s the terminology used by the Big Three to refer to a mother dhow and its 
adjoining pirate dhows. Typically, a mother dhow is a larger vessel, possibly a Boston 
whaler, used as the logistical hub to extend the longevity in time and distance of the 
piracy mission. Pirate dhows are often attached to the mother dhow with the boarding 
ladder, which is an aluminum ladder (Figure 4) (Agence France Presse [AFP], 2009). 
When pirates have located a vessel of opportunity, the pirates will use marine equipment 
such as grappling hooks to board the vessel. Pirates onboard can be equipped with 
satellite phones, global positioning system (GPS), rocket-propelled grenade launchers 
with warheads (RPGs), and AK-47s (Houreld, 2010). Commercial vessels traveling at 


slow speeds and having low freeboards are typically easier targets of opportunity. 
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Figure 4. Picture on the left is a mother dhow for logistical support at sea and three 
pirate dhows attached for the attacking of a commercial vessel (From Operation 
Ocean Shield, 2010). The picture on the right is a pirate dhow and pirates (From 
Ocean Shield, 2010). 


In the past, pirates have not been known to kill the crew of the pirated vessel. On 
February 22, 2011, pirates captured and killed four Americans onboard the Quest, a 58- 
foot yacht (Wadhams, 2011). These killings sparked questions about the effectiveness of 
the military in deterring piracy. Hillary Clinton, U.S. Secretary of State, stated, "We've 
got to have a more effective approach to maintaining security on the seas, in the ocean 


lanes that are so essential to commerce and travel" (Wadhams, 2011). 


D. OTHER POTENTIAL PIRACY TOOLS 
1. United Kingdom Global Model 


As part of the Big Three, the UK has its own forecasting model, the UK Global 
Model (UKGM), for "ease of maneuver and to optimize precision weapons" (Ritchie, 
2010). This model is run four times per day and provides as output a forecast for the 
following six days. Wave Watch III provides the wave characteristics and the 
Forecasting Ocean Assimilation Model-Nucleus of European Modeling of the Ocean 
(FOAM-NEMO) provides environmental information such as_ salinity and sea 
temperature for the model. The UKGM, with inputs from other systems, can be used to 
provide advice for likely enemy courses of action (COAs), but this model does not 


incorporate the INTEL about PAG CONOPS. 


1] 


De Technical Cooperation Program Maritime Group 


In October 2008, Australia, Canada, New Zealand, the UK, and the U.S. agreed to 
develop a piracy-based toolset that would support the efforts to counter the increasing 
threat of piracy with specific attention paid to the HOA. This toolset is the Technical 
Cooperation Program Maritime Group (TTCP MAR). Discussions held during a TTCP 
MAR meeting in September 2009 in Halifax, Canada, addressed the desire to "conduct a 
Red Teaming Exercise to help predict how pirate tactics and strategies may adapt and 
evolve as their current operations are impacted by coalition interdiction efforts" (D. 
DaSilva, personal communication, June 11, 2010).? The developers of the TTCP MAR 
were interested in learning about the author's research and how it could enhance TTCP 


MAR's development. 


E. EXPERIMENTAL DESIGNS 
1. Design of Experiments 


Design of experiments (DOE) has an extensive background and a variety of 
applications in the operations research and simulation community. Sanchez (2005) 
describes DOE as the ability of "the analysts to examine many more factors than would 
otherwise be possible, while providing insights that could not be gleaned from trial-and- 
error approaches or by sampling factors one at a time" (pp. 69). DOE can be thought of 
as an enabler that when properly implemented helps to reduce correlation among input 
variables while providing a robust design that eliminates duplication in the data analysis, 


which leads to greater statistical power. 


As described by Kleijnen, Sanchez, Lucas, and Cioppa (2005), DOE provides 
effective and efficient results without using a "trial-by-error approach" (pp. 263). 


Kleijnen et al. (2005) defines basic objectives analysts should have in developing their 


? TTCP MAR information was provided by D. DaSilva (UK civilian contractor for the CMF 
assessment team) to the author as a result of the author's operations research experience tour to the CMF in 
Bahrain on June 11, 2010. 
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DOE prior to formulating the meta-model. Kleijnen et al. (2005) recommends that the 
DOE must be an integral part in the developmental stages of multivariate polynomial 
modeling and should continue thereafter: 
DOE can uncover detailed insight into the model’s behavior, cause the 
modeling team to discuss in detail the implications of various model 
assumptions, help frame questions when the analysts may not know ahead 
of time what questions should be asked, challenge or confirm expectations 


about the direction and relative important of factor effects, and even 
uncover problems in the program logic. (pp. 266) 


When analyzing the output, analysts should be aware of nonlinear effects, 
discontinuous effects, chaotic behavior, and a simulation spin-up period (Kleijnen et al., 


2005). These model effects can lead to the wrong analytical result. 


Zz: Nearly Orthogonal Latin Hypercubes 


Depending on the question(s) to be answered, the modeler may be focused on 
orthogonality of input factors. | Orthogonality means the model's factors are 
uncorrelated—the cross product of any two input vectors in the model’s space is equal to 
zero. If there is multicolinearity among input factors, then the differences in standard 
error compared to the best-fit model may be extremely large and lead to inconclusive 


analytical results (Kleijnen et al., 2005). 


To provide a good DOE for many applications, Cioppa and Lucas (2007) develop 
an algorithm to provide nearly orthogonal Latin hypercube (NOLH) experimental 
designs. Cioppa and Lucas start with the goal of estimating simple yet common first- 
order effects in a model. In general, to estimate such a model, the number of simulation 


runs increases significantly with the number of factors, which may be prohibitive. Cioppa 
and Lucas define two objectives, Pmax and the condition number, cond (X "X ), that 
indicate how well the NOLH performs with respect to correlation: 


One measure, Pmax, gives the worst correlation between design matrix 


columns, whereas the other measure, cond (X‘X ), assesses the overall 
degree of nonorthogonality of the design matrix. A design matrix will be 
classified as nearly orthogonal if it has a maximum pairwise correlation 
no greater than .03 and a condition number no greater than 1.13. (pp. 47) 
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Other operations research studies that use DOE and the NOLH include Bodden 
(2010), who studies the survivability of the transformable craft; Abel (2009), who studies 
the effectiveness of frigate defensive measures in pirated waters; Wegner (2007), who 
studies system reliability and capabilities within a joint environment; and Joshua (2006), 


who studies the expansion of the NOLH as used in computer-based simulations. 


3. Design Points 


In modeling, specifically in DOE, input factors will affect the results generated by 
the simulation. An experimental design can be considered a mxn matrix where m 
defines the columns and n defines the rows of the matrix. The mcolumns will 
correspond to the input factors and the n rows will correspond to the factor settings 
during the number of runs. Each row represents a combination of factor-levels and is 
called a design point (Sanchez, 2006). In general, a proper DOE specifies a set of design 


points that efficiently enables the analysis after the experimental results are obtained. 
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HI. U.S. PIRACY MODELS 


In this chapter, the first generation Piracy Performance Surface (PPS) and the 
Next generation Piracy Performance Surface (PPSN) models are discussed, as well as 
how the PPS model is being used by the U.S. and allied nations’ militaries, and the 
differences between the two piracy models. This chapter also includes interviews with 


operators and users of the PPS model. 


A. FIRST GENERATION PIRACY PERFORMANCE SURFACE MODEL 


Just days after the MV Maersk Alabama, a U.S.-flagged ship, was hijacked by 
Somali pirates in the Indian Ocean on April 8, 2009, NAVO researchers were tasked by 
Rear Admiral David Titley, Oceanographer and Navigator of the Navy, to assist in the 
fight against piracy (B. Lingsch, personal communication, May 19, 2010).!° Admiral 
Titley wanted his staff to define how meteorologists could play a vital role in countering 
piracy operations, having knowledge of the limited military assets available to use and 
the impact of environmental conditions on the success of pirate activity.1! The PPS 
product was completed approximately two weeks after the Maersk incident, and has 


undergone a few revisions since its development. 


The primary purpose of PPS is to be used as a piracy-forward warning tool for 
military and commercial vessels. The model depicts the predicted pirate threat in SBR 
and GOA. Figure 5 shows the color coded output to the model. Red represents forecasts 
of high piracy threat against neutral vessels. Areas displayed in green have a lower 
piracy threat level. The Sixth Fleet Admiral and his staff are comfortable with the PPS 


convention in which red is indicative of areas more favorable for pirate activity.!? 


10 B. Lingsch is head of the INTEL Department at the Commander, Naval Meteorology and 
Oceanography Command (CNMOC) at Stennis Space Center in Mississippi. He is also the sponsor of the 
author’s thesis research. 


1! Dr. James Hansen and Mr. Bill Lingsch in a presentation to the Battlespace Atmospheric and Cloud 
Impacts on Military Operations (BACIMO) and Weather Impacts Decision Aids (WIDA) Conference on 
April 13-15, 2010. 


12 This information was gathered during the author's operations research experience tour on May 20, 
2010. The author was given a presentation by NAVO civilian contract Nathan Hooper. 
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The model's index is a weighted combination of forecast environmental suitability 
for pirate activity and historical observed pirate activity with weights determined by the 
Office of Naval Intelligence (ONI), of 90% and 10% respectively (N. Hooper, personal 
communication, May 20, 2010).!3 The environmental suitability is forecast using a wave 


model called the Shallow Water Near Shore model (SWAN). 


When a pirate event in the form of a boarding, attempted boarding, or suspected 
boarding occurs, an area surrounding the location is highlighted to represent a high-risk 
area in the PPS surface for the next forty-eight hours, then slowly dissipates over the next 
seven days. Such pirate events are reflected by the red dots in Figure 5. Historical 
attacks provided by ONI are also factored into the pirate activity index with a lesser 


contribution than the recent pirate activity. 
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Figure 5. PPS model output provided to the CMF assessment team that was based on 
April 12, 2010 at 1200 a.m. GMT. The areas encircled in red and annotated ‘high’ 
are areas that are more prone to pirate attacks, and thus should be areas avoided 
by neutral vessels and monitored by forces combating piracy in the SBR and 
GOA (From N. Hooper, personal communication, 2010). 


13 This information was gathered during the author's operations research experience tour on May 20, 
2010. The author was given a presentation by NAVO civilian contractor Nathan Hooper. 
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B. MILITARY APPLICATION 


As of this writing, the 1200 Greenwich Mean Time (GMT) PPS snapshot is 
briefed to the Commander of NAVCENT at the Combined Maritime Forces 
Headquarters' daily meeting.!* This snapshot of PPS is used to inform the chain of 
command of the current day’s relative pirate threat index. This 1200 GMT snapshot is 
also provided to the military forces deployed to the GOA and SBR in support of 
Operation Ocean Shield (OOS) and European Union Naval Force Somalia - Operation 
Atalanta. At the end of March 2011, PPSN will be briefed at this meeting. On March 1, 
2011, the name referencing the next-generation piracy model is the Pirate Attack Risk 


Surface (PARS). 


C. THE NEXT GENERATION PIRACY PERFORMANCE SURFACE 
MODEL 


The success of the PPS model led the Naval Meteorology and Oceanography 
Command (CNMOC) leadership to devote efforts to pursue a more advanced anti-piracy 
model, PPSN. Dr. James Hansen, Naval Research Laboratory (NRL) Monterey, was 


tasked with developing this next generation probabilistic forecasting tool. 


PPSN simulates pirate behavior within the METOC environment over a 72-hour 
forecast time period in 0.2-hour time steps with a probability surface generated at 12-hour 
time steps. The simulated behavior depends on intelligence parameters such as the 
PAG's camp location, the PAG's operating area, and the PAG's mission length. Figure 6 
displays the PPSN output, i.e. the relative forecast of pirate presence probability as a 


function of latitude, longitude, and time. 


The early version of the PPSN model simulated pirates from randomly generated 
starting locations. The pirates motored to a pre-determined location while being affected 
by environmental conditions (waves, winds, and currents), then drifted until the pirates 
ran out of simulated food, water, and/or khat, and then motored back to their beginning 


location. The output from the simulation was the relative probability that a pirate would 


14 This information was gathered on June 8, 2010 during the author's operations research experience 
tour. 
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be at a given latitude, longitude and time. PPSN represented probabilities with color 


coding, with the highest probabilities in red and the lowest probabilities in dark blue. 


latitude (degrees north) 
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Figure 6. The PPSN model output displaying the relative forecast of pirate presence 
probability as a function of latitude, longitude, and time. The two pictures 
displayed are the 24-hour forecast (left) and the 72-hour forecast (right) of relative 
pirate presence probability. Red corresponds to the highest probability forecast of 
pirate location. 


1. Modeling Differences Between PPS and PPSN 


Table 1 describes the modeling differences between the PPS and PPSN models. 
Overlay of waves, currents and INTEL | Simulation of pirate behavior. 
about pirate activity. 
Uses METOC forecasts of waves and | Uses METOC forecasts of winds, waves and 
currents. currents. 


Assumptions about environmental | Incorporates additional INTEL to include 
suitability based on a 9-foot skiff. pirate METOC knowledge, pirate skiff 
movement patterns, and pirate skiff speed. 
Updated every three hours unless pirate | In addition to updates if pirate activity 
activity occurs. occurs, the model takes operator inputs that 
drive the model to include pirate operating 
locations, length of time a pirate can 
operate, pirate camp locations, and observed 
pirate activity. 
Provides a _ snapshot forecast of | Time-integrates environment with pirate 
environmental suitability for pirate | CONOPS to produce relative forecast of 
activity. pirate presence probability. 


Table 1. Model comparisons between PPS and PPSN. 
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D. INTERVIEWS WITH OPERATORS AND USERS 


The author traveled to AFRICOM in Stuttgart, Germany, and met with the CMF 
assessment team in Bahrain to brief them on the PPS model and preliminary experiments 
using the MATLAB version of the PPSN model. The author met with users and 


operators to learn about their use of the PPS model, their needs, and about pirate tactics. 


In Bahrain, the operators, NATO and U.S. Intelligence officers, subject matter 
experts (SME), and high ranking military officers provided insight and direction that led 
the author to make recommendations to modify the PPSN model of pirate behavior. 
Based on these recommendations, several changes were incorporated in the new version 
(Python) of the PPSN model. Changes that were based on the author's recommendations 


included: !5 


e Defining factors in the PPSN model that INTEL and 
users justified as being significantly important in 
PAG CONOPS. 


e Implementation of different PAG search patterns 
(described below). 


e Redefining the factor definitions used in the model to 
assimilate the terminology being used by the U.S. 
and allied military operators. 


e Implementation of mapping PAG from their base 
location to their operating waypoint location. 


e Extending the run time of the model to weeks and 
months rather than days to reflect longer PAG 
CONOPS. 


13 A trip report of the author’s operations research experience tour is included in Appendix A of this 
thesis. 
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E. POST EXPERIENCE TOUR DESCRIPTION OF THE REDEVELOPED 
PPSN 


PPSN is a Monte Carlo probabilistic forecasting tool that takes as input 
environmental conditions, pirate CONOPS and gathered INTEL to produce as output the 
relative forecast of pirate presence probability as a function of latitude, longitude and 
time. PPSN is written in Python (x,y) version 2.6.5.1 (Python 2.6.2). The Python version 
of PPSN has been continually in development since June 2010, and later experiments 
were conducted with an updated version, finalized in October 2010, which is similar (but 


not identical) to the current operational version. 


As of the date of this thesis, PPSN is the only known operational Navy product 
that couples METOC and INTEL. It is used by the U.S. and allied forces to combat 
piracy in the SBR and GOA. 


1. Meteorology and Oceanography Conditions 


The METOC forecast used in the PPSN model is provided by NAVO 


oceanographic models. 


a. Oceanographic Models 


The Shallow Water Near Shore model (SWAN) provides the waves. 


The Coupled Ocean/Atmospheric Mesoscale Prediction System model 
(COAMPS) provides the currents. 


The Navy Coastal Ocean model (NCOM) provides the surface winds. 
b. METOC Parameters Provided by the User 


Input parameters describing the effect of METOC on PAGs are provided 
by the operators that drive PPSN. Typically, this information will be provided by the 
U.S. or the SMEs of the allied forces. These parameters affect the PAG's movement 
during transit from a base (camp) location to a waypoint location and when the PAG is 
conducting CONOPS in the waypoint operating area. The input parameters listed below 


are defined in more detail in Chapter IV of this thesis. 
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The factor by which the currents impact the PAG. 
The factor by which the winds impact the PAG. 


The threshold below which the PAG can operate under certain surface 
wind conditions. 

The wave height threshold below which the PAG can operate (analogous 
to wind threshold). 


Whether the PAG had knowledge of METOC conditions before leaving its 
base location. 


2. Pirate Action Group Mapping 
a. Base Mapping 


The PPSN model simulates PAGs, which is a group of pirate dhows that 
are attached to a mother dhow (a larger vessel) that is typically used as a logistical hub 
for extending distances from land and mission length. PAGs are given a specific 
beginning location (base) that can be a single point or a box, and may be a land-based 
camp or an origin point at a mother dhow at sea. Each base has a longitude and latitude, 
and a longitudinal and latitudinal size of the box, and an associated weight. The weight 1s 
normalized in the code and defines the likelihood that the PAG will use one base over 


another base 1n PPSN. 


b. Waypoint Mapping 


From the PAG's base, the PAG is mapped to a hunting area known as the 
waypoint location. The waypoint location can be a defined by a single point or a box. 
The waypoint location is given a longitude and latitude, and a longitudinal and latitudinal 
size of the box and a weight. The user can define the base and waypoint to be the same 
longitude and latitude. This could potentially be beneficial when mapping PAGs at 


distances 1,000 nautical miles from campsites. This technique was used in this thesis. 


C. PAG Mapping Demonstration 


Figure 7 displays an example of the overall mapping of PAGs from base 


locations to their respective operating waypoints (annotated by the arrow). Each 
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simulated PAG begins its mission at a location randomly sampled from the base 
distribution. For example, consider the third yellow box in Puntland off the HOA. This 
small yellow box is the starting base location that the operator may choose to model as 
the PAG's beginning location. The second yellow area labeled Distro2 may be the PAG's 
waypoint. The PPSN code requires the lower left-hand corner of each base and waypoint 
to be the longitude and latitude of a base or waypoint. The author used Google Earth to 
build the inputs for the mapping necessary for the PPSN model. 


When establishing Distro2, the user will define this box by longitude and 
latitude in degrees, (6.14 N ,64.86 E . The associated longitude and latitude size of the 


box is equal to five in this case. A size of five for a box means approximately a 
300x300 mile box. In this thesis, the author used a size of 0.01 for the base boxes and 


0.1 for the size of the waypoints boxes. 





Figure 7. Representation of the base and waypoint locations inputted by the user into 
the PPSN model for the appropriate mapping of each PAG. 


Z2 


a 


PAG Search Patterns 


This factor defines the movement pattern that the PAG will use once it has 


reached its waypoint longitude and latitude and begins the hunting phase (described in 


Phase 2 below). The PAGs can be given any of the following search patterns: 


4. 


QO (drift) - This value represents the PAG drifting as a function of the 
environmental conditions while searching for vessels of opportunity. 

1 (random walk) - This value represents the PAG randomly walking within 
the defined longitude and latitude waypoint while searching for vessels of 
opportunity. 


2 (zig-zag) - This value represents the PAG zig-zagging within the defined 
longitude and latitude waypoint. The zig-zag search pattern resembles the 
PAG moving across a known shipping transit lane while searching for vessels 
of opportunity. 


3 (transit) - This represents the PAG hunting only within the defined longitude 
and latitude waypoint. 


Three Phases of PPSN 


Three typical PPSN PAG CONOPS phases are as follows: 


I. 


Phase | is the PAG's transit from its base to its waypoint. During the transit, 
the PAG will be impacted by winds, currents, and wind and wave threshold. 
The PAG will continue to the defined waypoint until it gets within a 
tolerance (distance) or the PAG's mission length (defined in Chapter IV) has 
been exceeded. 


Phase 2 is the PAG’s hunting phase. During this phase, the PAG will be 
impacted by winds, currents, and wind and wave threshold. The PAG will 
hunt using one of the described search patterns until the PAG's remaining 
mission length is equal to the time it took the PAG to get to its waypoint 
location (transit time). If the transit time is greater than half the PAG's 
mission length, then the PAG will immediately return to its base. 


Phase 3 is the transit back to its base. During this phase, the PAG will be 
impacted by winds and currents, but will not be impacted by wind threshold 
and wave thresholds. 
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IV. VARIABLE DEFINITIONS 


This chapter describes the independent and dependent variables modeled in each 


design of experiments (DOE) in this thesis to analyze the PPSN model. 


A. INDEPENDENT VARIABLES 


The impacts of the following variables were studied using a NOLH design to 


produce a robust and space-filling DOE. A brief description of each of the independent 


variables is as follows: 


iL 


Number of pirate action groups (npi): This discrete factor represents the 
number of simulated PAGs, defined by the user, in the simulation. This 
factor must be an integer greater than zero. 

Number of bases (nbase): This discrete factor represents the number of 


distinct base locations the PAGs will originate in and must be an integer 
greater than zero. In this thesis, nbase represents the total number of land 
and sea bases. 


Mission length (t_mission ): This discrete factor represents the mean 
length of time in hours that PAGs remain active in the simulation. 


Number of PAG replications (numtracks): This discrete factor represents 
the number of times the model is replicated for every PAG in a given 24- 
hour period. For example, if numtracks equals 1,028 then the model will be 
run 1,028 times per PAG for each 24-hour period to generate the forecast of 
relative pirate presence probability. 

METOC knowledge (iweather ): This continuous factor represents the 
probability, iweather € [0,1], that PAGs have METOC knowledge prior to 


leaving their base. 


Skiff speed (skiff _ speed): This continuous factor represents the speed 


that a given PAG is expected to sustain during the transit and searching 
patterns. This factor is measured in knots and must be a positive real 
number. 


Drift current (drift _ curr): This continuous factor represents the factor by 
which the currents impact skiff movement, drift __ curr €[0,1]. 


Drift wind (drift_wind): This continuous factor represents the factor by 
which the winds impact skiff movement, drift __ wind €[0,1]. 
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Wind threshold (wind _thresh): This continuous factor represents the 
threshold below which PAGs can operate. If winds exceed a PAG's 
threshold, the PAG can no longer operate. The PAG will either return to its 
base or remain at its base if the PAG has not left. This factor is measured in 
knots and is a positive real number. 


10. Wave threshold (wave _thresh): This continuous factor represents the 


11. 


threshold below which PAGs can operate. If waves exceed a PAG's 
threshold, the PAG can no longer operate. The PAG will either return to its 
base or remain at its base if the PAG has not left. This factor is measured in 
knots and is a positive real number. 


Pirate search pattern (search _ pattern): ‘This categorical factor is an 


integer from zero to three that describes how the PAGs will conduct their 
search once they reach their waypoint operating areas. Table 2 describes the 
possible PAG search patterns. 


Search Pattern Description 


O (drift) 


PAG movement is determined by winds 
and currents only. 


1 (random walk) PAGs will be moving at the defined 


2 (zig-zag) 


3 (transit) 





Table 2. 


skiff _speed in a random walk, plus a 
component due to winds and currents. 

PAGs will be moving at the defined 
skiff _speed in a zig-zag pattern. This 


pattern represents PAGs moving in a 
Strategic motion east to west and west to 
east across a southerly to northerly transit 
lane. The zig-zag pattern will also be 
impacted by winds and currents. 

PAGs will be transiting at the defined 
skiff _speed and will be impacted by 


winds and currents. 


Descriptions of the four PAG search patterns. These patterns take effect 


when the PAGs reach their waypoint destination and begin to search for a vessel 
of opportunity. 
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B. DEPENDENT VARIABLES 


Summarizing and quantifying the output from the PPSN model was a challenge. 
Each simulation run produced a pirate density in each cell at multiple forecast leads 
(usually 12, 24, 36, 48, 60, and 72 hours). Therefore, the results in 43x51 cells at 


multiple forecast leads must be summarized. 


In addition, it was impossible to use historical pirate incidents to verify PPSN’s 
performance because historical METOC conditions and available INTEL corresponding 
to observed pirate activity have not been archived. In addition, there is insufficient 
INTEL gathered from historical pirate incidents within the SBR and GOA. Therefore, 
for any experiment conducted during this thesis, there is no definitive ground truth 


against which to compare results. 


Each simulation run in this thesis was summarized and compared usefully with 
the results from other design points to identify the variables that are most influential and 
have the greatest interaction with METOC conditions. Within each simulation, 
differences across the forecast lead times reflected sensitivity and interactions with 
METOC conditions. Differences in the probability fields across design points reflected 


the influence of the experimental variables. 


A brief description of each of the dependent variables is given below. 


Tt = forecast lead where tr = 0,6,12,...,7 


max 


T..., = longest forecast lead provided for the PPSN METOC 


x, = i" indexed longitude, where i=1,...,43 and x, =33°E and x,,, —x, =0.8° 


l 


y, = j" indexed latitude, where j =1,...,51 and y, =—15°N (15'S) and 


dp = design point number from the DOE 
r =replication number of each dp 


pirate _ prob(dp,T,i, ]) = PPSN-forecast of relative pirate presence probability as 
output, € [0,1] 





R = the set of comparable design points 


Zi 


r >, pirate _ prob(dp,t,i,j) €[0,1] 


mean _ pirate _ prob(t,1, j, R) = 
| | dpeR 

RMSD and MAXVAL are both output measurements used to measure the 

difference between the forecast of relative pirate presence probability for each design 


point and the mean pirate probability that is generated by the simulation. 


(1) Root mean squared difference (RMSD(dp,t, R)) is the mean square root of 


the squared difference between the cell-by-cell relative pirate probability 
pirate _ prob(dp,T,i, j), and the mean forecast of relative pirate presence 


probability, mean _ pirate _ prob(t,i, j,R), for the set of comparables R. 






43 51 


——)_) (pirate _ prob(dp,t,i, j) mean _ pirate _ prob(t,i, j,R)) 


RMSD(dp,t,R) = 
Dt) = lie 


i=l j=l 


(2) Maximum value (MAXVAL(dp,t,R)) is the cellwise maximum absolute 


value of the difference between’ the _ relative’ pirate probability, 
pirate _ prob(dp,t,i,j), and the mean forecast of relative pirate presence 


probability, mean _ pirate _ prob(t,i, j,R). 
MAXVAL(dp,t,R)= jmax | pirate _ prob(dp,tT,i, ]) mean _ pirate _ prob(z,1, j,R) 


i=l,...,43, j=1,...,51 


These inner variants measure the differences across forecasted relative pirate 
presence at different lead times within the same design point. The following equations 


will define the inner variants of RMSD and MAXVAL. 


Tmax 





l 12 
inner _mean _ prob(dp,i, j,T .) =—— >. pirate _ prob(dp,12:-t,i, j) 


max f=1 


12 
The inner variants are designed to capture the sensitivity of a given design point's 
probability field to METOC conditions, using the forecasts at different lead times as a 
proxy for changing METOC conditions. 
(3) The inner variant of RMSD (iRMSD(dp,T,,.,)) is the square root of the 


maximum squared difference between the cell-by-cell relative pirate 
probability, pirate _ prob(dp,t,i,j), and the inner mean _ relative pirate 


probability, inner _ mean _ prob(dp,i, j,T 


max 


), of each design point over available 


forecast leads (c's). 
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43 51 
ee > max ( pirate _ prob(dp,t,i, j)—inner __mean _ prob(dp,i, j,Tyax )) 


iRMSD (dp, Ty,x ) = aes r 
"SE jay jay SOP MAx 


(4) The inner variant of MAXVAL (iMAXVAL(dp,t,,,)) is the maximum 


cellwise absolute difference between a probability field and the inner mean 
relative pirate probability, inner __mean _ prob(dp,i, j,T.,...), of each design point 


max 


over available forecast leads. 


IMAXVAL(dp, Trax) = MAX | pirate _ prob(dp,t, j,i) —inner __mean _ prob(dp, J,1,Tpyax )| 


T<T Ax 


Smoothed variants of RMVSD, MAXVAL, iRMSD, and iMAVAL were also used. 
In the smoothed variants, each cell's value (pirate _ prob) was replaced by the average 


of the value plus the eighth adjacent cell's values (smoothed _ pirate _ prob). 
i+] jl 
smoothed _ pirate _ prob(dp,tT,i, j) =— » > pirate _ prob(dp,t,k,m) 
k=i-1 m= j-1 
Two other measures were also calculated. Mean 50" percentile (MAREA ) is the 
average area containing 50% of the relative forecast of pirate presence probability 


density. MAREA and the smoothed variant (MAREAs ) were not used in the analysis as 


discussed in Chapter VI, Section B. 
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V. SIMULATION EXPERIMENTAL DESIGNS 


This chapter describes the PPSN experiments used in this thesis: (A) computer 


memory requirements, (B) significant factors, and (C) variability across design points. 


This thesis uses the Air Force Research Laboratory Department of Defense 
Supercomputing Resource Center (AFRL DSRC) High Performance Computing (HPC) 
system, hawk, to perform the PPSN simulation runs. The AFRL DSRC HPC is located at 
Wright-Patterson Air Force Base in Ohio. The SGI Altix 4700 (hawk) consists of 18 
computer nodes with each computer node having 512 processors, all running a variant of 
the the Linux operating system (AFRL DSRC, 2010). Each of 16 computer nodes has 1.5 
GB of useable available memory while two computer nodes have 3.5 GB of useable 


memory (S. Upton, personal communication, February 24, 2011). !® 


This memory limitation significantly impacted the number of design point runs 
that were completed. Steve Upton (personal communication, February 24, 2011) 
calculated that 24,000 computer-hours were used to complete this thesis on hawk. Upton 
(2011) stated, 

This may seem inflated; in order to use the large memory nodes this thesis 

needed to "reserve" eight processors even though the PPSN model was 

using one processor in each of the jobs. In addition, jobs that did not 

complete in the allotted time were still charged with that amount of 


computer time. There were also a few trial and error missteps by the hawk 
user. 


A. MEMORY REQUIREMENT DESIGN OF EXPERIMENTS 


In attempts to run the PPSN model, the computing memory requirements were 
often exceeded resulting in failed design point runs. Preliminary experiments revealed 


that increasing t_mission, numtracks, and npi significantly increased the computer 


memory requirements of the PPSN runs, and high memory requirements could cause 


16 This information was provided by Steve Upton, SEED Center for Data Farming research associate. 
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early termination of the simulation. Therefore, an experiment was designed to analyze 
each of the three memory-driving independent variables separately to determine their 


influence on the PPSN output. 


1. Mathematical Calculation Used to Determine the Computer Memory 
Requirements 


Close examination of the code showed that the following equation approximates 
the required memory for each design point (S. Upton, personal communication, February 


24, 2011).!7 

memory _ estimate = ( pirate _ location _ size+ METOC _ array _ size) +25 

The division by 2.5 in the equation above is indicative of the compression ratio on 
Microsoft Windows software. 


ntaus = the number of forecasts 


max(t _ mission) 
SS ——— a + ntaus 


2*max(t__ mission) 
hw =| —-—_—_ 
24 


METOC _array _ size = (3 * 201 *171% 2* hrs ) +(2*321* 213% 2* hrs ) 


The values 171, 201, 273, and 321 are fixed array sizes based on the array size of the 
provided METOC data. 


pirate _ location _ size = npi x c (nw (numtracks ca 1)) + 24 x *) A (one) a (24 *(nw+t »)| 


dt t mission 





Using this memory calculation allowed the author to create design of experiments 


that could be run in a reasonable operational time and produce relevant output results 


with less than the approximately 24 GB (16*1.5GB) of memory available. 


17 This formula was provided by Mr. Steve Upton, research associate for the SEED Center for Data 
Farming. 
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De Description of Three Independent Variables That Increase Computer 
Memory Requirements 


The memory capacity of about 24 GB precluded using a space-filling design that 


varied t_ mission , numtracks , and npi throughout realistic ranges. 


Appendix B displays the values given to each design point. This appendix also 
shows the three cases modeled in this DOE. Design point one is used as the base case 
design point for this DOE (refer to Appendix B). Design point one will remain constant 
through the simulations and when analyzing the three memory requirements variables. 


Design point one has the following values associated with each variable: t _ mission = 36, 
numtracks =128, and npi=4. The computer memory requirement for this base case 


was less than | GB. 


a. The Number of PAG Replications 


The first sub-design, design points | and 2-16, varies numtracks while 
holding t_mission and npi constant. The input values given to numtracks vary 
between 128 and 2,048 PAG replications per 24-hour period in increments of 128 PAG 
replicates. This variable is used in the PPSN model to produce the forecast of relative 
pirate presence probability. This variable is not reflective of gathered INTEL from the 
military assets countering piracy in the SBR and GOA. It is solely used for modeling 


purposes. 


b. Mission Length 


The second sub-design, design points | and 17-35, varies t_ mission 
while numtracks and npi are held constant. The input values given to t_ mission vary 


between 24 and 336 hours in increments of 6, 12, and 24 hours, respectively (refer to 


Appendix B). Despite 336 hours being the longest t_ mission in this DOE, the CMF 


assessment team INTEL officer, LCDR K. Lutz, (personal communication, June 8, 2010) 
stated that the increase in pirate capabilities has enabled the PAGs to sustain pirating 
missions for longer than 336 hours; thus, 336 hours is a moderately low mission length 
for advanced PAG CONOPS. 
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c. The Number of PAGs 


The third sub-design, design points 1 and 36-46, varies npi while 
t_ mission and numtracks are held constant. The input values given to npi vary 


between 4 and 48 in increments of four PAGs. The current operational testing being 
conducted on PPSN has injected real-time INTEL reflecting the value associated with 


this variable. 


B. SIGNIFICANT FACTORS DESIGN OF EXPERIMENTS 


The following DOE with 33 design points investigates which PPSN input 
parameters are the most influential in determining the distribution of relative risk of 
pirate presence in PPSN and which PPSN input parameters interact most strongly with 


METOC conditions and INTEL parameters. 


Table 3 shows the variables modeled and the ranges of values given to each of the 


variables discussed in Chapter IV. With the exception of t_ mission , these values are 


realistic according to information gained during the author's operations research 


experience tour. 


npi 10 30 
t mission 

numtracks 

skiff _ speed 

drift _ curr 

drift _ wind 

wind _ thresh 

wave _ thresh 


iweather ( probability ) 





search _ pattern(categorical) 0,1,2,3 


Table 3. NOLH maximum and minimum values for the ten independent variables 
used in the significant factors design of experiments. 
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In addition, skiff _ speed standard deviation was set to 20% of each design 
point's skiff _ speed. The wave _thresh standard deviation was set to a NOLH design 


variable with values ranging from 0.1 to 0.9 multiplied by the value of wave _ thresh . 


Since npi ranges from 10 to 30, the PAGs are assigned to the six bases randomly. 


One-third are assigned to land bases, which are sampled without replacement until all 
land-base positions have been used at least once, and then with replacement until all 
PAGS have been assigned a base. Each land base 1s paired with a hunting waypoint. The 
remaining sea-based PAGs are also assigned to the three sea bases in the same way. For 


sea-based pirates, their base is identical to their hunting waypoint. 


A NOLH was used to achieve near-orthogonality and a good space-filling design 
for the 10 independent variables. Appendix C provides the full NOLH design of the 33 
design points. Due to the memory and runtime requirements, a rotational design, or other 
larger designs, was not feasible for this experiment. The runs which completed indicated 
that runtime was highly correlated with memory requirements in GB (refer to Figures 8 


and 9). 


Figure 8 represents the original DOE where the memory requirements for the 16 
design points that did not complete (level zero) ranged from 0.7 to 15.8 GB. These 
design points are 3, 5, 6, 11, 12, 14, 15, 16, 17, 18, 21, 24, 25, 26, 27 and 30. Design 
point 18 has the highest total memory requirement, 15.79 GB. This design point also has 
a drifting search _ pattern. For the design points that completed (level one), design 
points | and 2 are the outliers with memory requirements of 14.7 GB and 14.3 GB, 
respectively. This difference between these two design points and design point 18 is their 


search _ pattern. Design points | and 2 had a transit only search _ pattern. 
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Figure 8. Original DOE with 33 design points. A level of one indicates design points 
That ran to completion and zero indicates design points that failed to complete. 


In order for the remaining 16 design points to run to completion, the DOE was 


modified (Figure 9). The length of t_ mission was decreased by 20% of its original 


value, numtracks was decreased by 60% of its original value, and the allotted runtime 


was increased from 30 hours to 60 hours. Decreasing t_mission and numtracks 


reduced the remaining design points' memory requirements to less than 9.5 GB. 
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Figure 9. Modified DOE with 33 design points. The design points that did not complete 
(evel O) had a reduction in t_ mission and numtracks . 
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The scatterplot matrix (Figure 10) shows that the 10 independent variables used in 
the significant factors DOE have nearly linear independence, even after the modification 


of the DOE. The correlation between the variables is less than 0.3. 


Scatterp|ot Matrix 


searck — iweather «ave thresh wind drift wird drift curr skif speed 
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t mission skiff speed = drift_cum — drift_wind _ thresh h iweather pattern npi 


Figure 10. Scatterplot matrix of the significant factors DOE. 


C. VARIABILITY ACROSS DESIGN POINTS 


Long runtimes prevented conducting a larger experiment with many replications; 


hence, a small experiment was conducted to evaluate the variability inherent in the PPSN 
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model. The question that this experiment addresses is how much variation is there for a 
single design point due to the stochastic variation in the model. A single design point 
from the significant factors DOE is chosen—design point 13. This design point is chosen 
because it 1s not an outlier and its memory requirement is less than 3.5 GB, 0.9 GB to be 
precise. Design point 13 is replicated 23 times with the variable values as defined in 


Table 4. 


npi 16 
t mission 

numtracks 

skiff _ speed 

drift _ curr 


drift _ wind 
wind _ thresh 
wave _ thresh 


iweather 
search _ pattern 





Table 4. Design point 13 values for the variability across design points DOE. 
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VI. DATA ANALYSIS 


The previous five chapters discussed the problems of piracy, the two piracy 
performance surface models, the independent and dependent variables used in this thesis, 


and the simulation DOEs. 


In this chapter, the results from each of the experiments are analyzed. The 
conclusions drawn from this analysis are used for refining INTEL inputs and PPSN 


developmental efforts. 


A. METHODOLOGY AND TOOLS 


For each forecast lead time and design point, a picture in the form of a jpeg file is 
produced, which shows the forecast of relative pirate presence probability (see Figure 6 


for examples in Chapter HI, Section C). 


if Analytical Tool 


JMP is used as the statistical analysis software in this thesis.!° JMP's robust tool 
sets allow for the use of one software program to apply the following analytical methods 
used here: (1) partition trees, (2) stepwise regression methods, (3) sensitivity analysis, 


and (4) multivariate correlation. 


2. Linear Regression Analysis 
This thesis will use the following multivariate regression equation for determining 
the "best" fit model: 


k k ea 
y=fy +> Bx, +> Bi% +> > 8 4%; Té 
i=l i=l 9 


yal 4S1 
where € =error. 
Montgomery, Peck, and Vining (2006) discuss three stepwise-type procedures for 
analyzing data. These broad categories include forward selection, backward elimination, 


and stepwise regression. This thesis uses forward selection which assumes no 


18 JMP* , Version 9. SAS Institute Incorporated, Cary, N.C. Retrieved from http://www.jmp.com. 
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independent variables are initially in the model. The first variable to be included in the 
model is the independent variable with the greatest correlation to the dependent variable. 
Additional independent variables are added in the model if their partial correlation is 


significant after adjusting for the effects of the included independent variables. 


The significance levels for variables to enter and exit the regression model are 


Panter = 9-05 and P,,,- =0.10. Devore (2009) states the 


P -value (observed significance level) 1s the smallest level of significance 
at which H, would be rejected when a specified test procedure is used on 


a given data set. Once the P-value has been determined, the conclusion 
at any particular level @ results from comparing the P-value to a: 


l. P—value< a= reject H, at level a. 
2. P—value>a= donot reject H, at level a (pp. 313). 


The null hypothesis, H,, is the hypothesis assumed to be true, and the test procedures are 


an attempt to disprove the null hypothesis. Devore (2009) states "the two possible 
conclusions from a hypothesis-testing analysis are then reject H, or fail to reject H," 


(pp. 285). 


The coefficient of determination, R’ , is used as one of the key measures of model 
effectiveness (Montgomery et al., 2006). R* is the proportion of total variance explained 
by the model. A R* value close to 1 means most of the variation in the dependent 


variable is explained by the regression equation. 


3. Classification and Regression Trees 


The use of classification and regression trees can be a powerful tool for 
understanding results graphically. The tree methods explain the variation of the output 
variable (dependent variable) by splitting or partitioning on the data recursively, creating 
homogeneous branches until a stopping criterion is met. The stopping criteria for this 


thesis use the sum of squares for continuous variables and the log-likelihood chi-squared 
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for discrete and categorical variables.!? A regression tree is used for this thesis to show 


clear-cut partitions for the influences of the independent variables modeled in PPSN. 


B. MULTIVARIATE CORRELATION 


Chapter IV described each of the ten output metrics. The multivariate correlation 
scatterplot, depicted in Table 5, shows the pairwise correlations among the dependent 
variables resulting from the significant factors experiment. The high correlation among 
RMSD, MAXVAL and their smoothed versions indicates that only one of these four 
dependent variables needs to be used in the analysis of this thesis. Thus, RMSD is the 
first dependent variable used in the analysis. Furthermore, high correlation exists among 
IRMSD and iMAXVAL and their respective smoothed versions. Therefore, iRMSD is the 


second dependent variable used in the analysis. 


Table 5 also shows that MAREA and MAREAsS are negatively correlated to each 
of the other eight dependent variables and positively correlated to each other. However, 
these dependent variables did not significantly depend on any of the DOE input variables. 


Therefore, these output variables are not used in this thesis. 


19 Details on how JMP calculates the Pp — value using partition tree analysis can be found at 
http://www.jmp.com/blind/whitepapers/wp_montecarlo_cal.pdf. 
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Scatterplot Matrix 


_|iis_RMSD] ~ / 


‘F/ 
of: 
| 


0.0010.004 0.001 0 0.050.1 00.020.05 00.030.07 





Table 5. Multivariate correlation scatterplot matrix of the output statistics for the 
24-hour forecast from the significant factors experiment with 33 design points. 
This experiment was used to test the correlation between the output metrics. 
RMSD and iRMSD are the two dependent variables used in this thesis. 

c. RESULTS 


1. Computer Memory Requirement Experiment 


The computer memory requirement experiment analyzes the three memory 


variables, t_ mission, numtracks , and npi, to pinpoint how these variables affect the 
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PPSN output. The underlying question that this experiment answers is whether the value 
given to these variables alters the forecast of relative pirate presence probability. The 
information gathered from this experiment is important for the operational testing being 


conducted on PPSN.29 


a. Mission Length 


The modelers are using a mission length of seven days (168 hours) in 
operational testing being conducted on PPSN. D. Laleijini2! stated, "We want to keep the 
PAGs in the model for as long as we reasonably can based on processing time and 
memory requirements" (personal communication, February 7, 2011). The following 


t mission analysis will show why special precautions should be taken when decreasing 


t__mission to meet operational objectives. 


Figure 11 shows the results of varying t_ mission values together with the 
best fit quadratic equation and reveals that t_ mission does matter. What Figure 11 does 
not show is what specific values of t_ mission should be used. It may appear that the 
knee of the curve lies between a t_ mission value of 150 and 225 hours. This is an 
incorrect assumption. These results do not show that values of t_ mission produce the 
best results. These results show that small values of t_ mission will produce very 
different results than increased values. A ft mission value that reflects true PAG 


CONOPS should be used rather than a t_ mission that gives PPSN a shorter run time. 


The military operator conducting missions in the SBR requires an accurate 
portrayal of the relative pirate presence probability. Unlike PPS, PPSN provides better 
information that focuses the military's surface assets to a specific area within the SBR. 
Since the SBR covers millions of square miles using realistic CONOPS for simulated 
PAGs is important. Decreasing a PAG's mission length will produce very different PPSN 


output results. 


20 Effective March 1, 2011, the name of the next- generation Piracy Performance Surface model, 
PPSN, was changed to the Pirate Attack Risk Surface model, PARS. 


21H, Lalejini is a civilian researcher for NRL-Stennis Space Center in Mississippi. 
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Figure 11. Effect of varying t_mission on RMSD for the 72-hour forecast probability 
field together with the best fit quadratic model. The model's R’ is 0.57. 


b. Number of Pirate Action Group Replications 


The number of PAG replications, numtracks, is another variable that 
significantly impacts computer memory requirements. The variable numtracks 
determines how many times the PPSN model is replicated with the same input variables 


to produce the pirate probability fields. 


Figure 12 shows the result of varying numtracks. A smaller value of 
numtracks produces a larger difference between each design point and the mean 
probability. Figure 12 shows that RMSD 1s influenced by numtracks over the entire 


experimental range from 128 to 2048. 
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Figure 12. Effect of varying numtracks on RMSD for the 72-hour forecast probability 
field together with the best fit quadratic model. The model's R* is 0.757. 


Initially, the operational testing being conducted on PPSN was using 
numtracks equal to 512. After a discussion with the developer and modelers, numtracks 


was increased to 1024 replications. 


In closing, this variable should be hard-coded into the PPSN model rather 
than letting the operator set the value since it is influential but it is not impacted by 


changing METOC or INTEL. 


C. Number of Pirate Action Groups 

The current operational testing being conducted on PPSN uses a value of 
npi that is based on recent anti-shipping activity messages from the National Geospatial- 
Intelligence Agency (NGA) indicating when a PAG location has been reported (D. 
Lalejini, personal communication, February 7, 2011). The observed PAGs remain in the 
PPSN model for a maximum of six days. A typical value is npi =6, which is lower than 
most values used in the significant factors experiment of this thesis. Those conducting 


the operational testing include only reported PAGs, but they are currently unable to 
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acquire the most accurate and real-time intelligence on current PAG activity in the SBR 


and GOA. The true number of PAGs operating at one time is believed to be greater. 


Figure 13 shows npi ranging between 8 and 48 PAGs. The author's 


Operations research experience tour interviews reveal that limited knowledge is available 
on the exact number of PAGs operating within (on the high seas) the SBR and GOA and 


along the coastline at camp sites. For this reason, a wide range of scenarios for npi 
should be considered. Figure 13 shows that PPSN output is sensitive to the value of npi 
throughout the experimental range, but the effect of npi diminishes substantially around 
npi=30. At this point, it should be noted that for all design points in this thesis 


experiment, the same four bases and four hunting waypoints are used for all values of 


npi, so differences across design points are due to the number of PAGs, not the number 


or locations of bases and hunting waypoints. 





Figure 13. Effect of varying npi on RMSD for the 72-hour forecast probability field 
together with the best fit quadratic model. The model's R* is 0.944. 


In conclusion, this variable should be controlled by the operator. Specific 


intelligence should be gathered as to provide realistic values for this variable. 
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pe Significant Factors Experiment 


The significant factors experiment is analyzed using partition trees and linear 
regression. The ten independent variables this DOE uses were selected based on 
information collected during the author's operations research experience tour to the 


Combined Maritime Force's headquarters in Bahrain. 
The DOE answers the following two questions: 


1. Which PPSN input parameters are the most influential in determining the 
distribution of relative risk of pirate presence in the PPSN? 


Z Which PPSN input parameters interact most strongly with METOC 
conditions and INTEL parameters’? 


Information gathered from this DOE will help the developer, modelers, and 
operators in focusing their efforts towards a more direct, systematic approach when using 
PPSN and gathering INTEL on PAG CONOPS. 

a. Significant Factors Using Partition Trees 


In a partition tree, the splits indicate the best predictors of the output 
metrics. The partition tree for RMSD shows which METOC and INTEL variables have 
the greatest effect on the PPSN output, while the z~RMSD partition tree shows how 


variables interact with METOC changes across the forecast leads. 


Figure 14 shows that the majority of the variation in RMSD 1s explained 


by search _ pattern. Recall, the following search _ pattern definitions: 
e 0 (drift) 
e 1 (random walk) 
e 2 (zig-zag) 


e 3 (transit) 
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All Rows 
Count 33 LogWorth Difference 


Mean 0.0028967 8.7601614 0.00479 
Std Dev 0.0025349 










C | search_pattern(2, 1,0 


Count 27 LogWorth Difference 
Mean 0.002025 2.0028082 0.00103 
StdDev 0.000737 


<—_|search_ pattern(3) _[> 


Count 6 
Mean 0.0068198 
Std Dev 0.0039657 









drift_wind>=0.2 

Count 22 LogWorth Difference 
Mean 0.0018335 15271973 0.00062 
StdDev 0.0004755 


drift_wind<0.2 
Count 5 


Mean 0.0028673 
StdDev 0.0011203 
















drift_wind<0.6128 drift_wind>=0.6128 

Count 17 LogWorth Difference |} Count 5 
Mean 0.0016931 16050846 0.00051 |] Mean 0.0023108 
StdDev 0.0004452 StdDev 0.0001609 


C{search_patterni)> {earch_pattern(0, > 


Court 8 || Count 9 
Mean 0.0014209 |] Mean 0.0019351 
StdDev 0.0003412]] StdDev 0.0003931 










Figure 14. Partition tree displaying a R° value of 0.58. The dependent variable is 
RMSD where t =72 hours 


One of the important results of the author's operations experience tour was 
the addition of search _ pattern in the PPSN code. The developer initially only had the 
drifting search pattern; three additional search patterns were implemented. Figure 14 
shows that search _ pattern is highly significant in influencing the PPSN output. The 


INTEL community should concentrate their efforts on determining PAG search behavior. 


This knowledge will prove to be beneficial for the PPSN output. 


The variability chart for RMSD at 72 hours (Figure 15) shows 
search _ pattern equal to 3 (transit only) and reflects the greatest difference in variability 
in the PPSN model. This means that a PAG modeled with a transit only searching pattern 
has the greatest difference in pirate probability fields compared to the mean pirate 
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probabilities. As stated previously, specific INTEL should be gathered to determine if 


PAGs use a transit only searching pattern when looking for a target vessel. 


Variability Chart for RMSD-72hr 
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Figure 15. Variability chart for RMSD at 72 hours 


The second partition occurs on drift_wind at 0.2. Recall that 
drift__wind is a continuous variable between O and | that represents the factor by which 


winds impact the PAGs' movement. Another way to look at this is that it is important to 
know how much of PAGs' movement is affected by surface winds in order to predict the 
relative distribution of pirate presence probability. The third partition in Figure 14 also 


occurs on drift_wind. This time the split occurs on a factor of 0.6128. This result 
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means that drift_ wind above 0.2 and below 0.6128 leads to output similar to the mean 
pirate probability fields, but drift_wind below 0.2 or above 0.6128 produces results 


different from the mean pirate probability fields. 


The inner metric, iRMSD, captures the sensitivity of the probability field 
within each design point to a change in the forecast leads. Figure 16 shows a partition 
tree using iRMSD as the dependent variable. The first and second splits occur on 
search _ pattern indicating that a search pattern of 3 (transit) or O (drift) makes the 
design point more sensitive to METOC. The third split occurs on wind _ thresh. Recall 
that wind _ thresh is a continuous factor that represents the wind speed threshold below 
which PAGs can operate. If winds exceed a PAG's threshold, the PAG will either return 


to its base or remain at its base 1f it has not left. 


INTEL should be gathered for this specific variable depending on the 
change in the METOC across each season. Since this variable describes much of the 
variability in RMSD, associating a realistic value is important. A recommendation is to 


have the modelers at ONI input the wind threshold and standard deviation. 


The final split occurs on npi, the number of PAGs simulated in PPSN. As 


described previously in Section C.1 of this chapter, better intelligence should be gathered 


on the value given to this variable. 


50 


All Rows 
Count 33 LogWorth Difference 


Mean 0.0027323 5027492 0.0049 
StdDev 0.0029667 








< [search_patiemsy> 

oun 6 
Mean 0.0067398 
StdDev 0.0053524 






C |search_pattern(2, 1,0) 


Count LogWorth Difference 
Mean 0.0018417 3.2245413 0.00124 
StdDev 0.0008908 















aan EERE —_—— 
@ search_pattern(2, 1) search_pattem(0 


Count 21 LogWorth Difference Count 6 
Mean 0.0015662 25039032 0.00105 Mean 0.002806 


StdDev 0.0007421 StdDev 0.0006991 







wind_thresh>=23.13 wind _thresh<23.13 


Count 14. LogWorth Difference || Count f 
Mean 0.0012149 0.4983909 0.00046 || Mean 0.0022689 
StdDev 0.0004525 StdDev 0.0007266 


npi>=22 npi<22 
Count 7 


Mean 0.0009831 || Mean 0.0014467 
StdDev 0.0003029||StdDev 0.000476 


Figure 16. Partition tree displaying a R° value of 0.47. The dependent variable is 
iRMSD . 


b. Significant Factors Using Linear Regression 


Linear regression is the second way this experiment's results are analyzed. 

This thesis uses linear and quadratic terms, and first-order interaction terms as shown in 
Chapter VI, Section A.2 in the linear regression equation. 

(1) RMSD as a Measure of Output. The linear regression 

model's summary of fit (Figure 17) shows that the model generated by the stepwise 

regression has a R* value of 98.4 percent. This R* is nearly 1, meaning that most of the 


variation in the model is explained by the regression equation. 
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Summary of Fit 


RSquare 0.983951 

RSquare Adj 0.97297 

Root Mean Square Error 0.000417 

Mean of Response 0.002897 

Observations (or Sum Wogts) 33 

Analysis of Variance 

Sum of 

Source DF Squares Mean Square _ F Ratio 
Model 13 0.00020232 0.000016 89.6068 
Error 19 0.00000330 1.737e-7 Prob>F 
C. Total 32 0.00020562 <.0001* 


Figure 17. Summary of fit and analysis of variance for the linear regression analysis for 
RMSD. 


Figure 18 shows that seven of the ten independent variables 
modeled in this DOE are significant in determining RMSD for a forecast lead of 72 
hours. Notice that unlike the partition tree, drift_ wind is not included in the linear 
regression model. This result is a product of the partition tree recursively splitting on 


homogeneous groups while the linear model uses a regression equation. 


Also, take note of the interaction between search _ pattern and 


iweather. Recall, iweather 1s the probability that PAGs have METOC knowledge prior 
to leaving on a mission. This result is interesting because PAGs are given a specific 


search _ pattern and iweather probability before the simulation spin-up. Both of these 
variables are highly influential. Therefore, operators should focus their efforts on getting 
valid INTEL about these two variables. A PAG having knowledge of METOC is going 
to mitigate the effect of the PAG's search pattern on the forecast of relative pirate 
presence probability. 

In Figure 18, the two terms with quadratic effects are t_ mission 


and iweather. These two squared terms show that these two variables have a nonlinear 


effect. The variable iweather should be estimated based on intelligence gathered of the 
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sophistication at each camp site. This variable does interact with search _ pattern, the 


variable that was shown in the partition tree to describe most of the variability in the 


model. 


Another interesting aspect of Figure 18 is that the three computer 
memory requirement variables discussed earlier in this chapter are significant. As 
discussed previously, care must be taken when selecting values for these three variables 


as they are highly influential. 


Sorted Parameter Estimates 


Term Estimate StdError tRatio Prob>|t| 
(iweather-0.50061)*(search_pattern{0&1&2-3}-0.63636) -0.007244 0.000426 -17.01 <.0001* 
search_pattern{0&1&2-3} -0.002174 0.000137 -15.83 <.0001* 
iweather 0.0023018 0.000253 9.10 <.0001* 
(wind _thresh-25.0012)*(search_pattern{0&1&2-3}-0.63636) -0.000463 5.612e-5 -8.24 <.0001* 
wind _ thresh 0.0001575 2.786e-5 5.65 <.0001* 
(search_pattern{0&1&2-3}-0.63636)*(numtracks-345.424) -3.728e-6 6.653e-7 -5.60 <.0001* 
(t_mission-139.697)*(t_mission-139.697) 3.5577e-7 7.052e-8 5.05 <.0001* 
t_mission -0.00001 2.183e-6 -4.57 0.0002* 
(skiff_speed-20.0012)*(wind _ thresh-25.0012) -9.844e-6 2.958e-6 -3.33 0.0035* 
skiff_speed -2.622e-5 8.848e-6 -2.96 0.0080* 
npi -3.658e-5 1.36e-5 -2.69 0.0145* 
numtracks -1.343e-6 5.031e-7 -2.67 0.0151* 
(iweather-0.50061 )*(iweather-0.50061 ) 0.002382 0.001055 2.26 0.0360* 





Figure 18. Model parameter estimates overview for RMSD at 72 hours. 


(2) IRMSD as a Measure of Output. The output metric, 
iRMSD , captures the sensitivity of a given design point to METOC conditions, using the 
forecasts at different lead times as a proxy for changing METOC conditions. Figure 19 
provides the sorted parameter estimates; this reveals the independent variables that are 


significant and influential. 


One interaction term that has not been seen in any of the previous 


analyses 1s skiff _ speed and wind _thresh. Interactions among variables can often be 


difficult to interpret. When analyzing interactions, one area of focus is each variable's 
parameter or /. The £ value allows the analyst to determine the effect of each variable 


and their interaction on the model. 


For example, this linear model includes the following terms: 


p wind _ thresh Nvind _ thresh os skiff _ speed X skiff _ speed ai wind _ thresh:skiff _ speed Nvind _ thresh N skiff _ speed 
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This equation implies that the effect of a change in skiff _ speed is dependent on the 
value of wind _thresh. Likewise, the effect of a change in wind _ thresh is dependent 


on the value of skiff _ speed . 


Figure 19 shows that t_mission appears as the only nonlinear 
term (other than interaction terms). As stated previously, the value of t_ mission 1s 


important when modeling PAG CONOPS in the PPSN model. 


Sorted Parameter Estimates 


Term Estimate Std Error tRatio Prob>|t| 
(iweather-0.50061)}*(search_pattern{041&2-3 1.63636) -0.0170073 O.000375 -26.71 =000T 
iweather 0.0031655 0.00021 15.06 =00074* 
(t_mission-139.697)*(t_mission-139.697) 6.9561e-7 458645e8 11.90 = 0001 
search_pattern{0&1&2-3} -0.001473 0.000138 -10.65 = 0001 
(wind _thresh-25.0012)*(search_pattern{0&1&2-3}-0.63636) -0.00043 4.887e-5 -8.79 =0001 
(skiff_speed-20.0012)*(wind_thresh-25.0012) “0.000017 2721e6 -6.21 =0004* 
(search_pattern{0.41&2-3}0.63636)*(npi-20.0606) -8.066e5 O.000018 -4.47 0.0004" 
(t_mission-139.697 )*(iweather-0.50061) O.0000316 7.184e-6 4.40 0.0004" 
(npi-20.0606)}*(numtracks-345.424) 3.2898e7 7.597e8 4.33 0.0005* 
wind _thresh 95219e5 2.37185 4.02 0.0070" 
skiff_speed 26788e5 7.41206 3.61 0.0023" 
npi 0.0000324 0.000011 2.94 0.0095* 
(skiff_speed-20.0012)*(numtracks-345.424) -1559e-7 5 698e8 -2. 74 0.0147" 
drift_wind O.0007921 0.000339 2.34 0.0323" 
t_mission 3.6294e-65 1.645e-6 221 0.0424" 
numtracks 1.048e8-6 4 84e-7 2.17 0.0455" 


Figure 19. Significant factors sorted model parameter estimates. R* 





3. Variability Experiment 


The variability experiment is designed to see how the variability for a single 
design (design point 13) point replicated 23 times compares with the variability across 
design points in the significant factors experiment. Table 6 shows the ratio of the 


standard deviation of RMSD for the variability experiment (0,,,) to the standard 
deviation of RMSD for the significant factors experiment (Og¢ racrors) for three 


forecast leads. For each forecast lead, the ratio is less than 16%. This metric provides 


evidence that the variation in the significant factors experiment is primarily due to the 


variable settings in the DOE, rather than the variation created within the PPSN 
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simulation. So, for example, if one operator were to run the simulation with the same 
input variables and values more than once, the operator should achieve similar forecasts 


of relative pirate presence probability. 


O VAR 


O sIG _ FACTORS 


24 hour RMSD 
48 hour RMSD 
72 hour RMSD 





Table 6. The ratio between the standard deviation of design point 13 from the 
significant factors experiment and the 33 design point significant factors 
experiment. The output measurement is RMSD for forecast leads of 24, 48, and 
72 hours. 


The results of three carefully designed simulation experiments identify the most 
important drivers of the PPSN output and their significance. The results of this thesis 
have also provided insights that have been and can be used to improve parameter 
selection for operational implementation of the PPSN model to allow quick turnaround of 
updates to the product in response to new intelligence, within memory constraints, and 


without sacrificing product quality. 


5D 


THIS PAGE LEFT INTENTIONALLY BLANK 


56 


Vil. CONCLUSION AND RECOMMENDATIONS 


A. RESEARCH SUMMARY 


This thesis research includes visits and interviews with users of PPS and other 
counter-piracy forces to understand the operators’ needs and to gain intelligence on the 
most current pirate CONOPS within the SBR and GOA. This gathered intelligence led to 
recommended changes in the PPSN?2 code. In addition, using robust experimental 
designs, this research identifies the most significant INTEL factors driving the PPSN 
output. This gathered information is being used by the developer and modelers for the 
refining of these input variables and choosing the best functionality for enhancements to 
the PPSN model. This research also unveils input variables that are influential in the 
computing memory requirement and runtime. This information is being used to focus 
efforts on setting these variables at realistic levels, without sacrificing the model's 
efficiency and effectiveness, to allow for quick turnaround of updates to the PPSN model 


in response to gathered INTEL. 


B. RESEARCH QUESTIONS 


This thesis addresses the following questions: 


1. | What improvements can be made to the operational version of the PPSN 
model that will enhance the product’s output? 


2. Which PPSN input parameters can be adjusted to decrease computing 
memory requirements and run time? 


3. Which PPSN input parameters interact most strongly with METOC 
conditions? 


4. Which PPSN input parameters are the most influential in determining the 
distribution of relative risk of pirate presence in the PPSN’? 


22 Effective March 1, 2011, the name of the next- generation Piracy Performance Surface model, 
PPSN, was changed to the Pirate Attack Risk Surface model, PARS. 
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To answer Questions 2, 3, and 4 above, a systematic approach, not a trial-by-error 
approach, to experimental designs is used. The DOE use a robust design and use a 
realistic model of PAG CONOPS that conveys operators’ knowledge about current pirate 
behavior as seen in the SBR and GOA. 


The memory requirement findings in this thesis identify the three key input 
parameters that impact computer memory requirement, which is also highly correlated 
with runtime. These variables are the PAG's mission length, the number of PAG 
replications in the simulation, and the number of PAGs. The analysis by the author 
concludes that the values inputted for each of these variables should be considered with 
the utmost care as their associated values greatly impact the forecast of relative pirate 
presence probability. For the PAG's mission length, the value associated with this input 
variable should be based on current PAG CONOPS and not based on computing 
constraints; for example, if a PAG has the ability to be at sea for up to four weeks, then 
the mission length should be set accordingly. Setting the input value to anything less will 
sacrifice validity in the PPSN output. It 1s recommended that the number of PAG 
replications be controlled by the modelers as this input variable is not indicative of 
gathered INTEL or METOC, but rather, specific to the PPSN code. The last variable, the 
number of PAGs, should be modeled based on INTEL, but in the event that insufficient 
INTEL is available then the operator should input postulated PAGs into locations of 
typically high pirate activity. 


The search pattern the PAG uses when attempting to locate a target vessel of 
Opportunity is the most significant factor in the PPSN model. This variable was 
introduced into the PPSN code because the author gained intelligence on this variable and 
informed the developer of its significance. Operators and military members combating 
piracy should attempt to acquire INTEL on the specific search pattern that PAGs use in 
the SBR and GOA. 


Another significant variable that is driving the simulation is the factor by which 


the wind impacts the PAG's movement. The author recommends that this variable be 
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controlled in the model by METOC and piracy subject matter experts in the SBR and 
GOA as they have a greater knowledge in this area. The value given to this factor does 


impact the output. 


An interesting input variable in this thesis research is the probability that PAGs 
have METOC information prior to leaving their base location. In the linear regression 
model, this variable is significant and shows a strong interaction with the PAGs' search 
patterns and the PAGs' mission lengths when analyzing the sensitivity of the design 
points to a change in METOC across the forecast lead times. Proper INTEL should be 
gathered on the specific sophistication of PAG camp sites. The value given to this 
variable is highly influential and interacts with the most important variable, 


search _ pattern. 


The results of this thesis did not come at an easy cost. The code was broken in 
different ways, resulting in modifications by the developer to make PPSN better and 
more robust, which delayed the completion of experiments for several months. The 
software required to run the PPSN model was not available on the Hamming shared 
cluster at the Naval Postgraduate School, thus, this thesis required the use of high 
performance computing power using the hawk from Wright-Patterson Air Force Base in 
Ohio. During the simulation runs of this thesis, some design points would require longer 
than 60 hours of computing time, and there were even times when this runtime was 
exceeded, which led to a failed design point run. Computer memory calculations show 
that design points that were typically below 3.5 GB ran to completion in 30 hours. To 
ensure that all the design points ran to completion, this thesis required a reduction in the 


PAG mission lengths and the number of PAG replications. 


C. RECOMMENDATIONS 


The following recommendations from this thesis research are as follows: 


1. The PAGs’ mission lengths and the number of PAGs should be given 
realistic values despite their impact on computing memory requirements and 
runtime. Altering these values to unrealistically low values will have a 
detrimental impact on the model. 
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2. The PAGs' search patterns are highly influential and interact most strongly 
with METOC conditions, so INTEL should focus on _ accurately 
understanding PAGs' search strategies. 


3. Better INTEL on the number of PAGs needs to be provided to the modelers. 
This information will result in improved forecasts of relative pirate presence 
probability. 


D. THESIS IMPACT 


The author's experience tour resulted in a complete overhaul of the PPSN code by 
the developer. The author's sponsor, Bill Lingsch, who is head of the INTEL Department 
at CNMOC, stated, 


Piracy in the Somalia over the last two years has dramatically increased 
impacting worldwide commercial shipping. As good practices are 
implemented by mariners to minimize the piracy threat the pirates tactics, 
techniques, and procedures (TPs) are evolving. In the past there has 
been anecdotal evidence of how and when pirates would attack. The 
thesis work done under [the author] applies the much needed scientific 
rigor to understand what variables really matter in attempting to predict 
where and when pirates are most likely to attack. These results are huge 
as Navy moves forward in building a forecast model for areas of highest 
probability of attacks. Task Force Commanders can then use this product 
to determine where to allocate limited assets in a vast area to deter and/or 
interdict pirates. [The author's] thesis work provides the groundwork for 
both the operational product and where to focus follow on research. Her 
engagement throughout the operational and scientific community to 
address piracy has truly accelerated support to the operational Navy in 
addressing this growing problem. (personal communication, March 4, 
2011) 


The after action report provided by the author resulted in changes made to the 
existing PPSN model. While this research was being conducted the PPSN code was 
being reworked and altered. This created an intense working environment and a 


challenge to the forward progress necessary for this research. 


After a year of dedicated hard work on helping to provide the military leaders and 
operators with a piracy tool, the author is receiving a master's degree from the Naval 
Postgraduate School in Operations Research a week after the PPSN model becomes 
operational. This is a landmark in the history of military analysis as it is the first 
operational prediction product that couple METOC and INTEL. 
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E. FUTURE RESEARCH 


Since this thesis was the first systematic exploration of the PPSN simulation, an 


abundant amount of follow-on research exists. Some of the future research relating to 


piracy can extend to the following areas: 


I. 
Di 
3. 


Differentiating between neutral vessels and PAGs. 
Providing an operational risk management matrix for military assets. 


Guidance for the proper allocation of resources (ships, air, unmanned 
systems) to cover a large body of water. 


Extension to the issues surrounding drug trafficking on the high seas. 


Using PPSN to reduce uncertainty about the location of camp sites once a 
vessel has been pirated. 


Game theory approach to optimize the PAG's best course of action and 
responses. 


Finding the means to validate the model against historical data. 
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APPENDIX A. © AUTHOR'S OR EXPERIENCE TOUR 


The Naval Postgraduate School operations research experience tour occurs during 
the last six weeks of the 5th quarter. During this time, the operations research student 
takes time out of the rigors of the previous four quarters to gain information in the form 
of intelligence but more commonly, data, to be used for their thesis research. The author 
of this thesis was afforded a very unique experience tour. The author traveled to Stennis 
Space Center in Mississippi; Paris, France; Stuttgart, Germany; Bahrain and finished 


back in Monterey. 


The first stop in the author's experience tour was to Stennis Space Center along 
the Mississippi Gulf Coast where the author briefed the sponsor, the Naval Meteorology 
and Oceanography Command. During this time, the author was provided invaluable 


information on the PPS model and its background. 


Paris, France, was the next stop in the author's experience tour. The author 
presented and participated in the NATO Modeling and Simulation Group (MSG-088) 
conference. The lead U.S. Chairman for this group is Dr. Gary Horne?3, Naval 
Postgraduate School Operations Research professor. Dr. G. Horne stated, 

The context of the Task Group is to work on NATO related issues. The 

Task Group will look to use Data Farming in the 6 realms: High 

Performance Computing, Visualization of simulation data output, Rapid 

prototyping of scenarios, Design of experiments, Collaborative processes, 

and Model Development. Each of the members that have participated in 

the MSG will be a subgroup to help solve the designated problem. 

(personal communication, May 26, 2010) 

At the MSG-088 conference, the author did not gather information on piracy, but 


was able to gain information on different techniques used by other countries to solve 


sophisticated problems. 
The third stop in the author's operations research experience tour was to the 
United States African Command headquarters in Stuttgart, Germany. This visit was vital 


23 Dr. G. Horne is also a professor in the Operations Research Department at the Naval Postgraduate 
School in Monterey, CA. 


63 


for the sharing of information on the current PPS model and the future PPSN model, 
along with describing how the author's thesis research would impact and enhance the 
development of the PPSN model. At the time of the author's experience tour, AFRICOM 
had little to no partnership with Naval Postgraduate School. As a result of the author's 
experience tour, a new relationship was formed between this command and the Naval 


Postgraduate School, specifically with the Operations Research department. 


After the author's presentation, Lieutenant Commander Netherlands Navy (OF 3) 
J. Bertelink2* provided positive feedback and recognized the possible benefits of using 
the PPSN model for anti-piracy operations conducted by the Netherlands Navy. 
Lieutenant Commander Bertelink (personal communication, September 7, 2010) stated, 

One of my duties is to organize and (af possible) improve the METOC 

support to the NLD navy units. As you probably know the NLD navy 

plays an active roll (siz) in the NATO Operation Ocean Shield and the EU 

Operation Atalanta, both anti piracy operations in the Somalia basin. 


Therefore your presentation about the PPS-next model looked very 
interesting to me and I would like to ask you some questions about it. 


The final stop of the author's experience tour was to the Combined Maritime 
Forces (CMF) headquarters in Bahrain. The briefing room reserved for the author was 
filled with a multitude of high-ranking NATO and EU military members who were 
concerned with the increase in pirate activity within their area of concern, the SBR and 
GOA. They were familiar with the PPS model, but skeptical and yet intrigued by the 
PPSN model that would supersede PPS. The presentation was filled with questions that 
helped the author understand what tools the operators were requesting. During the 
remainder of the author's time with the CMF Assessment Team, the author was able to 
gain important intelligence on current military operations and pirate definitions, as well 


as CONOPS that occur within the SBR and GOA. 


The author, now equipped with a wealth of knowledge from the previous stops of 
the experience tour, returned to the Naval Postgraduate School in Monterey for the final 


stop. The author briefed the findings of the experience tour to the developer, modelers, 


24 T ieutenant Commander Netherlands Navy J. Bertelink is head of the Royal Netherlands Navy 
Command, Hydrographic Office. 
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and sponsor, which led to the redevelopment of the PPSN model. This version of the 
PPSN model written in Python (x,y) will be the operational model used in combating 


pirate activity in the SBR and GOA. 


65 


THIS PAGE LEFT INTENTIONALLY BLANK 


66 


APPENDIX B. MEMORY REQUIREMENT DOE 


Appendix B shows the variables values according to each design point and their 


respective base to waypoint mapping input values. 


t_mission varies according to design point 
npi varies according to design point 
numtracks varies according to design point 


1 
0.3 






skiff_speed 

drift_curr 

drift_wind 

wind_thresh (mean) 
wave_thresh (mean) 
iweather (mean) 
search_pattern (categorical) 
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APPENDIX C. 


SIGNIFICANT FACTORS DOE 


point threshold | threshold pattern 
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